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Description 

BACKGROUND OF THF frlVfflyrriQM 
5 1. Field of the Invention 

[0001] This invention relates to the field of computer graphics systems. More particularly, this invention relates to 
generation of high resolution stereo 3D images in a head tracked stereo display system. 

"> 2- Art Background , 

[0002] The human eye can be modeled as an optical system coupled to a retina, with the retina functioning as a 
light transducer. The human eye is immersed in physical space filled with light rays. A point source of light exists at 
every point in the physical space that is a boundary between transparent and opaque surfaces, or a boundary between 
i5 transparent surfaces of different refractive indices. Human stereo vision is achieved by immersing the two eyes at dif- 
ferent locations within the physical space. 

[0003] A head tracked stereo display system simulates the interaction of human eyes with the light rays of the phys- 
ical space in order to enable a viewer to perceive 3D images. A head tracked stereo display system senses the location 
in physical space of the viewer's head and eyes, computes a pair of stereo images of a virtual object based upon the 

20 location of the head and eyes, and generates the stereo image on a stereo display device. 

[0004] A head tracked stereo display system requires that only two images be computed and generated at a time, 
which results in substantially less computation than holographic systems. Moreover, the pair of stereo images have the 
same appearance as a hologram. Wrth a head tracked stereo display system, the virtual object appears to remain sta- 
tionary when the viewer's head tilts, or when the viewer's head moves to look around the side or over the top of the vir- 

25 tual object. For further discussion regarding head tracked stereo display systems, refer to Paley, W.B. Head-trackino 
Stereo Disol av. Techniques and Applications. Proceedings of SPIE, Feb 1992. 

[0005] However, the range of head movement available to the viewer in previous head tracked stereo display sys- 
tems is limited because the stereo display device remains stationary as the viewer's head moves. As the viewer's head 
moves around to the side of the virtual image rendered on the stereo display device, the virtual image becomes clipped 

so due to the angled position of the viewer's head in relation to the display surface of the stereo display device. 

[0006] Moreover, previous head tracked stereo display systems cannot intermix light from the virtual objects and 
light reflected from real objects in order to enable a viewer to perceive a natural arrangement for the real and virtual 
objects. If the viewer perceives the stereo images while facing the stereo display device, physical objects positioned 
between the stereo display device and the viewer's head block the light from the stereo display device and interrupt the 

35 virtual image. 

[0007] As will be described, the present head tracked stereo display system generates complex 3D stereo images 
to enable a greater range of head movement for a viewer. Moreover, the present head. tracked stereo display system 
intermixes light virtual and real objects to enable a viewer to perceive a natural arrangement for the real and virtual 
objects. 

40 

Summary of the Invention 

[0008] A method and apparatus is disclosed for generating high resolution 3D images in a head tracked stereo dis- 
play system. The present method and apparatus enables a wide range of viewer head movements in a head tracked 
45 stereo display system. The present method and apparatus enables a viewer to perceive virtual objects superimposed 
over real objects for a wide range of viewer head movements. The present method and apparatus tracks viewer head 
movement and automatically adjusts display configuration and stereo viewing pipelines to enable a wide range of 
viewer head movement in a head tracked stereo display system. 

[0009] The present method and apparatus also enables a viewer to perceive real objects and virtual objects in a 
so natural arrangement. A viewer is able to perceive real objects positioned in front of virtual objects in three dimensional 
space. Moreover, the viewer is able to perceive virtual objects positioned in front of real objects in three dimensional 
space. The present method and apparatus renders real objects in a virtual space, and generates a z buffer for the real 
objects in order to intermix the light reflected from the real objects and light from virtual objects. 
[001 0] The present head tracked stereo display system employs a display device disposed within a base housing. 
55 The display device generates stereo images rendered by a graphics rendering system implemented on a computer. A 
mirror is coupled to rotate around the display surface of the display device, such that the mirror transmits the stereo 
images to a viewer. A tracking device coupled to the computer senses the movements of the viewer's head in three 
dimensional space. The computer controls the angular position of the mirror with a motor to transmit the stereo images 
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to the viewer as trie viewer moves. The image rendering system uses the angular position of the mirror to adjust the 

viewing matrices for reflection of the stereo images by the mirror. 

[0011] To intermix the light from real and virtual objects, the computer maintains a data base of the location and 
dimensions in three dimensional space of the real objects. The image rendering system generates a z buffer indicating 
s three dimensional positioning for the real objects. The reaJ objects are rendered with no color. A haH silvered mirror is 
positioned to transmit the stereo images to the eyes of a viewer, and transmit tight reflected from the real objects to the 
eyes of the viewer. An LCD array panel selectively blocks transmission of the light reflected from the real object to the 
eyes of the viewer under control of the computer, such that the light reflected from portions of the real object obscured 
by the virtual object are blocked. 

10 

Brief Description of the Drawings 

[0012] 

is Figure 1 illustrates a computer based system for generating graphic images and responding to a user's input in 
accordance with the teachings of the present invention. 

Figure 2 illustrates an arrangement of major programs and storage areas contained within the memory 16 illus- 
trated in Figure 1 . 

20 

Figure 3 illustrates an example display plate coordinate system, which has its origin at the lower left hand corner 
of the visible portion of a display surface of the stereo CRT 

Figures 4a - 4d illustrate a rotating mirror apparatus for generating head tracked stereo images of virtual objects 
25 to enable a viewer to move around and perceive various perspective views of the virtual objects. 

Figure 5 shows an alternative embodiment of a rotating mirror apparatus for generating a 3D head tracked stereo 
image superimposed over a real object. 

30 Figure 6 illustrates a head tracked stereo display system that enables a viewer to perceive an arrangement of vir- 
tual objects and real objects such that real objects obscure virtual objects and virtual objects obscure real objects 
in a natural manner. 

Figure 7 shows an arrangement of virtual objects and real objects generated by the head tracked stereo display 
35 system of Figure 6. 

Detailed Description pf th e Invent i on 

[001 3] A method and apparatus is disclosed for generating complex 3D stereo images to enable a greater range of 
40 head movement for a viewer and to intermix light from virtual and real objects to enable a viewer to perceive a natural 
arrangement for the real and virtual objects. In the following description for purposes of explanation, specific applica- 
tions, numbers, apparatus and configurations are set forth in order to provide a thorough understanding of the present 
invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these 
specific details. In other instances, well known systems are shown in diagrammatical or block diagram form in order not 
45 to obscure the present invention unnecessarily. 

[001 4] Referring now to Figure 1 , an exemplary computer based system for generating graphic images and 
responding to a user's input in accordance with the teachings of the present invention is illustrated. A computer 10 com- 
prised of three major components is shown. The first of these is'an input/output (I/O) circuit 12 which is used to com- 
municate information in appropriately structured form to and from other portions of the computer 10. In addition, the 
so computer 10 includes a central processing unit (CPU) 14 coupled to the I/O circuit 12 and a memory 16. These ele- 
ments are those typically found in most general purpose computers. The computer 10 is intended to be representative 
of a broad category of computer systems. 

[001 5] A magnetic disk 20 is shown coupled to the I/O circuit 12 to provide additional storage capability for the com- 
puter 10. As is well known, the disk 20 may store other computer programs, characters, routines, images, etc., which 
55 may be accessed and executed by the CPU 1 4. It will be appreciated that additional devices may be coupled to the com- 
puter 10 for storing data such as magnetic tape drives, as well as networks which are in turn coupled to other computer 
systems. 

[001 6] A CD ROM 20 is shown coupled to the I/O circuit 12 to provide an additional storage capability for distribut- 
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ing pre computed 3D stereo images in accordance with the teachings of the present invention. Also, an LCD panel con- 
troller 28 is shown coupled to the I/O circuit 12. The LCD panel controller 28 receives bit map data from the computer 
10 to control the transparency of the pixels of an LCD panel array. 

[001 7] A motor control unit 30 is shown coupled to the I/O circuit 1 2. The motor control unit 30 enables the compu- 
5 ter 10 to control a servo motor. Alternatively, the motor control unit 30 enables the computer 10 to control a stepper 
motor. As will be described, the servo motor or stepper motor controlled by the motor control unit 30 enables the com- 
puter 10 to control positioning of a rotating mirror to generate virtual images. 

[0018J A raster display device 24 is shown coupled to the I/O circuit 1 2 and is used to display images generated by 
the CPU 14 in accordance to the teachings of the present invention. A wide variety of raster (or pixel mapped) display 

io devices may be utilized as display device 24. In the current embodiment, the display device 24 comprises a stereo CRT. 
As is described more fully below, the display images generated by the CPU 14 through the display device 24 are per- 
ceived by a viewer through a mirror. Therefore, the CPU 14 generates the display image so as to reverse the left and 
right positions of the pixels comprising the image. The mirror again reverses the left and right positions to enable the 
viewer to properly perceive the image. 

is [001 9} A keyboard 1 8 is shown coupled to the I/O circuit 1 2 and is used to input data and commands into the com- 
puter 10, as is well known. A pair of stereo shuttered glasses 120 is shown coupled to the I/O circuit 12. The stereo 
shuttered glasses 120 include a pair of embedded ultrasonic receivers 122 for receiving ultrasonic sound waves. The 
stereo shuttered glasses also have embedded an infrared controlled switch for controlling the shuttered lenses. 
[0020] A 3D 6-axis mouse 130 is shown coupled to the I/O circuit 12. The 3D mouse 130 includes a set of three 

20 ultrasonic receivers 132 for receiving ultrasonic sound waves. The 3D positions of the 3D mouse 130 and the shuttered 
glasses 120 are sensed by a 3D 6-axis head-tracker 140, which for one embodiment is mounted to the stereo CRT 24. 
The 3D ultrasonic tracking device 140 has embedded three ultrasonic transmitters 142 that generate the ultrasonic 
sound waves received by the ultrasonic receivers 122 and 132. 

[0021 ] Referring now to Figure 2. one arrangement of major programs contained within the memory 16 illustrated 
25 in Figure 1 is shown. In particular, there is shown a frame buffer 36. which serves as a pixel map of the display 24. The 
frame buffer 36 represents the video memory for the display 24, wherein, each storage location in the frame buffer 36 
corresponds to a left or right eye pixel on the stereo CRT 24. Thus, the frame buffer 36 comprises a two dimensional 
array of points having known coordinates corresponding to the pixels on the stereo CRT 24. The frame buffer 36 is pref- 
erably arranged as a double buffer, such than an image can be accessed from one buffer while another image is being 
30 displayed from the other buffer. 

[00221 The memory 16 also comprises a variety of programs executed by the CPU 10 that implement functions 
according to the teaching ol the present invention, as disclosed in this specification. Additionally, the memory 16 further 
comprises other programs for controlling or performing other well known functions and operation on computer systems. 
[0023] For one embodiment, the viewer wears the stereo shuttered glasses 120, and may manipulate the 3D 6-axis 
35 mouse 130. The 3D positions of the 3D mouse 130 and the shuttered glasses 120 are sensed by a 3D 6-axis head- 
tracker 140. The travel time of the ultrasonic sound waves, between the ultrasonic receivers 122 and 132 and the ultra- 
sonic transmitters 142, are used to triangulate the 3D positions of the shuttered glasses 120 and the 3D mouse 130. 
The 3D mouse 130 is used to manipulate a virtual image perceived by the viewer. 

[0024J In a head tracked stereo display system, a stereo viewing display pipeline is specified by two 4x4 perspective 
40 viewing matrices (effectively, one monocular pipeline for each eye). These matrices implicitly contain information about 
the overall physical configuration of the viewing. Each resulting matrix includes a skew component. 
[0025] The physical configuration of the viewing is determined by the location of the display surface in 3-space and 
the dynamic location of the viewer's eyes. When these parameters are known, the viewing matrices are unique to within 
a scale factor. The viewing matrix for each eye corresponds to a pyramid having a tip at the viewer's eye and a base 
45 defined by the four corners of the display window within the display surface. The front and back clipping planes are par- 
allel to the plane of the display surface, if the display surface is assumed to be perfectly flat. 

[0026] In the current embodiment, a computer graphics application specifies the relationship between physical 
coordinates and virtual coordinates (VC) by a matrix P. The relative position, orientation, and scale implied by the 
matrix P specify how the virtual and physical worlds are to be superimposed. (This scale factor is denoted g). The phys- 
so ical configuration of the stereo display device and the sensed real time location of the viewer's eye's contribute the 
remainder of the information necessary to the final 4x4 viewing matrices. The final 4x4 viewing matrices are used a 
parameters for the graphics rendering system implemented on the computer 10. 

[0027] To implement the teachings of the present invention, the stereo CRT 24 is regarded as having a physical 
coordinate system registered to its display surface. This physical coordinate system is hereinafter referred to as display 
55 plate coordinates (DPC). 

[0028] Referring to Figure 3, an example DPC coordinate system is illustrated. DPC has its origin at the lower left 
hand corner of the visible portion of a display surface of the stereo CRT 24. The x axis proceeds horizontally to the right. 
The y axis proceeds vertically upwards. The z axis is normal to the. virtual display surface, with positive coordinates out 
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towards the viewer. 

I0029] A window on the virtual display surface 50 is defined by specifying a lower left hand and an upper right hand 
corner for the window as two DPC points L and H on the z« O plane. Each of the viewers eyes has a separate coor- 
dinate in the DPC space. An example coordinate of a single eye point is denoted E. The front and back clipping planes 
are given by a distance F and B along the z axis. 

[0030] The view matrix P for a single eye, expressed in column vector matrix format, that maps visible points in 
DPC into the extended unit cube of [-1 + 1] [-1 +1] [-1+ 1], is as follows: 


1-B 7 
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[0031] The equations above apply to display devices such as the stereo CRT 24, as well as projection stereo dis- 
plays and stereo LCD panels. It should be noted that the intraocular distance, which is the distance between the 
viewer's eyes, is not directly represented in the view matrix P. 

[0032] In a head-tracking display system, parallax on the display surface is not necessarily horizontal. If a viewer 
observes the display with eyes oriented vertically, then the parallax at the screen will be completely vertical. The amount 
of parallax at the screen is not consistent even for a given head distance and object location. When a viewers head is 
turned 30° to one side of the screen, the parallax at the screen surface is less than when the screen is squarely faced. 
[0033] In traditional computer graphics, the viewing projection point is referred to as the "eye point" or "viewpoint" 
and is intended to correspond to the viewer's eye. However, for the purposes of accurate display, the viewpoint location 
must be accurately identified physiologically. In general optical terms, the viewpoint of a lens system is the first nodal 
point. 

[0034] Accurate viewpoint location for the nodal points of the eye can be achieved by using eye tracking hardware 
to acquire information about the direction of gaze of each of the viewer's eyes. The combination of gaze information and 
head and eye location can be used to accurately locate the eye first nodal points in the DPC space. In addition, the gaze 
direction information can be used for identifying the region of the display surface space that corresponds to the fovial 
portion of the retina, and thai deserves to be rendered with high spatial detail, 

[0035] Alternatively, errors due to uncertainty in eye first nodal point location can be minimized by anticipating the 
likely direction of the viewer's gaze. The likely direction of the viewer's gaze may be the center of the stereo window on 
the virtual display surface 50. When the 3D mouse 130 is employed, the viewer's gaze is likely to be in the direction of 
the "hot spot" of an interactive virtual image, as it is likely that the tightest accuracy requirement reflects "touching" the 
mouse to a virtual object. Choosing the direction of the viewer's gaze at the tip of the 3D mouse 130 compensates for 
errors due to rotation of the eyes. Moreover, the image rendering software implemented on the computer 10 maintains 
DPC coordinates for the "hot spot" of the stereo image displayed on the virtual display surface 50. 
[0036] The current embodiment of the present invention employs an ultrasonic tracking device 140 to acquire 
dynamic viewer head location and orientation within the DPC space. However, it should be noted that the present 
method does not depend on the tracking technology used. Head location and orientation data is used to derive the rota- 
tional centers of the viewer's eyes in real time using fixed vectors from the location of the ultrasonic tracking device 140 
to the viewer's eyes. 

[0037] The intraocular distance of a viewer's head can very between individuals by as much as ±2 cm. In the current 
embodiment, the fixed vectors from the ultrasonic tracking device 140 to the viewer's eyes are viewer specific in order 
to account for the intraocular distance of a given viewer. The fixed vectors also account for the registration of the head 
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tracking dance to the viewer's head. ■ 

[0038] In order for viewer to perceive computer generated objects as three dimensional physical objects, it is nec- 
essary to have a display frame rate sufficient for motion fusion. With stereo imagery there is the additional phenomenon 
of induced stereo movement, wherein objects displayed at a low frame rate appear to deform and twist. For further clis- 

5 cussion, refer to Tyler, C.W., Induced Stereo movement. Vision Res., Vol 14, 609-613, Pergamon Press, 1974. 

[0039] Figure 4a illustrates a rotating mirror apparatus 51 for generating head tracked stereo images of virtual 
objects. The rotating mirror apparatus 51 enables a viewer to move around and perceive various perspective views of 
the virtual object or objects. The stereo CRT 24 is embedded within a cylindrical shaped housing 58. The stereo CRT 
24 is substantially centered along the axis of the cylindrical housing 58. A display surface 55 of the stereo CRT 24 pro- 

w trudes through the upper end of the cylindrical housing 58. A mirror 54 is coupled to a rotating platform 56. For one 
embodiment, the mirror 54 forms a 45 degree angle with respect to the display surface 55 of the stereo CRT 24. 
[0040] For one embodiment, a stepper motor (not shown) causes the rotating platform 56 to revolve around the 
center axis 57 of the cylindrical housing 58. The stepper motor is coupled to the motor control unit 30. The motor control 
unit 30 is coupled to receive control information from the computer 10. The computer 10 causes the stepper motor to 

is control the angular position of the rotating platform 56. and thereby control the angular position of the mirror 54. 

[0041 ] Figure 4b illustrates the rotating mirror apparatus 51 for generating a 3D head tracked stereo image of a vir- 
tual object 210. The viewer 200 wears the shuttered glasses 120. The 3D position of the shuttered glasses 120 is 
sensed by the ultrasonic tracking device 140. For one embodiment the ultrasonic tracking device 1 40 is mounted to the 
rotating platform 56 because ultrasonic sound waves have limited range and require line ol sight paths between the 

20 ultrasonic tracking device 1 40 and the shuttered glasses 120. For an alternative embodiment, a tracing device having 
extended range and not requiring line of sight paths is mounted to the stereo CRT 24. 

[0042] For one embodiment the computer 10 senses the travel time of the ultrasonic sound waves between the 
ultrasonic transmitters 142 and the microphones 122. Based upon the travel time, the computer .10 triangulates the 3D 
positions of the shuttered glasses 1 20. The computer 1 0 corrects for the speed of sound given the air temperature and 

25 other factors to triangulate the 3D position of the shuttered glasses 1 20. 

[0043] The computer 10 tracks the 3D position of the head of the viewer 21 0 by tracking the 3D position of the shut- 
tered glasses 120. As the viewer 200 moves around the rotating mirror apparatus, the computer 10 adjusts the rota- 
tional angle of the mirror 54 such that the mirror 54 maintains an angular position facing the viewer 200. 
[0044] The computer 1 0 and the motor control unit 30 implement a stabilizing control process that limits movement 

30 of the rotating platform 56 to major head movements of the viewer 200. For one embodiment, the computer 10 and the 
motor control unit 30 implement dynamic Kalman filtering for stabilization control. 

[0045] As the computer 10 adjusts the rotational angle of the mirror 54 to track movement of the viewer 200, the 
computer control effectively causes the mirror 54 to follow the averaged location of the head of the viewer 200 over a 
short time interval. 

35 [0046] An image rendering system implemented on the computer 10 generates stereo images on the stereo CRT 
24. The stereo images on the display surface 55 of the stereo CRT 24 are reflected by the mirror 54. The mirror 54 
reflects the stereo images toward the viewer 200 wearing the shuttered glasses 120. The viewer 200 perceives the 
reflected stereo images as a virtual object 2 1 0. The virtual object 21 0 appears transparent to the viewer 200 if the minor 
54 is a half silvered mirror. If the mirror 54 is a solid mirror, the virtual object 210 appears solid to the viewer 200. 

40 [0047] To the viewer 200, the virtual object 210 appears to be generated by a reflection of the stereo CRT 24 posi- 
tioned behind the mirror 54. As the computer 10 rotates the mirror 54 to track the movements of the viewer 200, the 
reflection of the stereo CRT 24 moves within the DPC space accordingly. It will be appreciated that the reflection of the 
stereo CRT 24 appears to turn about the vertical axis 57 as the mirror 54 rotates. 

[0048] Figures 4c - 4d illustrate the changing positions of a reflection 60 of the stereo CRT 24 as the mirror 54 
45 rotates in order to track the movements of the viewer 200. A top view of the rotating mirror apparatus 51 is shown, 
including the rotating platform 56, the stereo CRT 24, and the rotating mirror 54. Also illustrated is a reflected display 
surface 50 of the slereo CRT 24 as perceived by the viewer 200 through the mirror 54. 

[0049] Figure 4c illustrates the position of the reflection 60 of the stereo CRT 24 for an example position of the 
viewer 200 in relation to the rotating mirror apparatus 51. Figure 4d illustrates the position of the reflection 60 of the 

so stereo CRT 24 for another example position of the viewer 200 in relation to the rotating mirror apparatus 51 . 

[0050] The computer 10 determines the 3D position of the reflection 60 of the stereo CRT 24 from the rotational 
angle of the rotating platform 56. The computer 10 adjusts viewing matrices corresponding to the eyes of the viewer 
200 in order to position the display surface of the stereo CRT 24 at the position of the display surface 50 of the reflection 
60 of the stereo CRT 24. Therefore, the DPC space is registered to the display surface 50 of, the reflection of the stereo 

55 CRT 24 as the virtual objects are mapped into the virtual space. Moreover, the computer 10 performs a top-bottom 
reversal of the coordinates of the stereo images on the display surface of the stereo CRT 24 to compensate for the 
reflection of the stereo images in the mirror 54. 

[0051] Figure 5 illustrates an alternative embodiment of a rotating mirror apparatus for generating a 3D head 
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tracked stereo image superimposed over a real object. In the example shown, the rotating mirror apparatus enables a 
viewer 220 to perceive a virtual image superimposed over a surgical patient 230. The virtual image is a 3D image of the . 
surgical patient derived from a computer aided tomography (CAT) scan. 

[0052] A stereo CRT 70 is rotatably coupled to a ceiling mounted track 74. The ceiling mounted track enables trans- 
s lation of the position of the stereo CRT 70 along two dimensions parallel to the ceiling plane. A half silvered mirror 72 
is mounted to the stereo CRT 70. For one embodiment, the half silvered mirror 72 forms a 45 degree angle with respect 
to the display surface of the stereo CRT 70. The computer 10 actuates a stepper motor (not shown) to rotate the stereo 
CRT 70 and the mirror 72 about an axis defined by the vertical shaft 75. 

[0053] The viewer 220 wears the stereo shuttered glasses 120. The ultrasonic tracking device 140 senses the 3D 
io position of the shuttered glasses 120. For one embodiment, the ultrasonic tracking device 140 is mounted to the stereo 
CRT 70. The computer 10 tracks the 3D position of the head of the viewer 220 by sensing the 3D position of the shut- 
tered glasses 120. As the viewer 220 moves, the computer 10 actuates the stepper motor to adjust the rotational angle 
of the stereo CRT 70 in order to maintain the mirror 72 facing the viewer 220. 

[0054] As the computer 10 adjusts the rotational angle of the stereo CRT 70 and the mirror 72 to track movement 
is of the viewer 220. a computer filtered control effectively causes the mirror 72 to follow the averaged location of the head 
of the viewer 220 over a short time interval. 

[0055] An image rendering system implemented on the computer 10 generates stereo images on the stereo CRT 
70. The stereo images on the display surface of the stereo CRT 24 are reflected by the mirror 72. The mirror 72 reflects 
the stereo images toward the viewer 220 wearing the shuttered glasses 120. The viewer 220 perceives the reflected 
20 stereo images as a virtual image of a CAT scan. The virtual image appears transparent to the viewer 220 since the mir- 
ror 72 is a half silvered mirror. 

[0056] To the viewer 220. the virtual image of the CAT scan appears to be generated by a reflection of the stereo 
CRT 70 positioned behind the mirror 72. As the computer 10 rotates the stereo CRT 70 and the mirror 72 to track the 
movements of the viewer 220, the reflection of the stereo CRT 70 moves within the DPC space accordingly. 

25 [0057] The computer 10 determines the 3D position of the reflection of the stereo CRT 70 from the rotational angle 
of the stereo CRT 70 around the vertical shaft 75 and from the x-y motion of the shaft. The computer 1 0 adjusts viewing 
matrices corresponding to the eyes of the viewer 220 in order to position the display surface of the stereo CRT 70 at the 
position of the display surface of the reflection of the stereo CRT 70. The DPC space is registered to the display surface 
of the reflection of the stereo CRT 70 as the virtual image is mapped into the virtual space. Moreover, the computer 10 

30 performs a top-bottom reversal of the coordinates of the stereo images on the display surface of the stereo CRT 70 to 
compensate for the reflection of the stereo images in the mirror 72. 

[0058] Figure 6 illustrates a head tracked stereo display system that enables a viewer to perceive an arrangement 
of virtual objects and real objects. The stereo CRT 24 is shown mounted in an inverted position such that stereo images 
generated on the stereo CRT 24 are reflected by a mirror 310. The mirror 310 is a half silvered mirror. A viewer 300 
35 wears the stereo shuttered glasses 120. The ultrasonic tracking device 140 (not shown) is mounted to the stereo CRT 
24. 

[0059] To the viewer 300, the stereo images on the stereo CRT 24 appear to be generated by a reflection of the 
stereo CRT 24 positioned behind the mirror 310. As a consequence, the computer 10 adjusts viewing matrices corre- 
sponding the eyes of the viewer 300 in order to position the display surface of the stereo CRT 24 at the position of the 
40 display surface of the reflection of the stereo CRT 24. Therefore, the DPC space is registered to the display surface of 
the reflection of the stereo CRT 24 as the virtual objects are mapped into the virtual space. Also, the computer 10 per- 
forms a top-bottom reversal of the coordinates of the stereo images on the display surface of the stereo CRT 24 to com- 
pensate for the reflection of the stereo images in the mirror 310. 

[0060] The computer 10 uses the ultrasonic tracking device 140 to sense the 3D position of the head of the viewer 
45 300. Thereafter, the image rendering system on the computer 10 generates stereo images on the stereo CRT 24 to 
render the virtual objects behind the half silvered mirror 310. The viewer 300 wearing the stereo shuttered glasses 120 
perceives the reflected stereo images as one or more virtual objects positioned behind the half silvered mirror 310. The 
half silvered mirror 310 also enables the viewer 300 to perceive a real object 330 positioned behind the half silvered 
mirror 310. 

so [0061] The computer 10 maintains a physical object data base indicating the 3D geometry of the real object 330. 
and indicating the physical location of the real object 330. The real object 330 is modeled as a black object, i.e. no color, 
in the physical object data base. The image rendering system of the computer 10 renders the physical object 330 in the 
virtual space in order to generate z buffer values for the physical object 330. 

[0062] if a physical object is positioned within the DPC space in front of a virtual object from the point of view of the 
55 viewer 300. the z buffer of the image rendering system of the computer 10 causes no light to be emitted from the stereo 
CRT 24 for portions of the virtual object obscured by the real object. In this manner, the viewer 300 perceives the virtual 
object to be behind the real object. 

[0063] On the other hand, if a physical object is positioned within the DPC space behind a virtual object from the 
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point of view of the viewer 300, an LCD array panel 320 is employed to obscure portions of the physical object that are 
behind the virtual object. The LCD array panel 320 is positioned over the half silvered mirror 310. The computer 10 con- 
trols the array of LCD pixels comprising the LCD array panel 320. The computer 10 causes the LCD array panel 320 to 
selectively block light from passing through the half silvered mirror 320 to the viewer 300. The computer 10 causes the 
s LCD array pane! 320 to block portions of the physical object that are behind the virtual object. As a result, the viewer 
300 perceives the virtual object to be in front of the real object. 

[0064] Figure 7 shows an arrangement of virtual objects and real objects generated by the head tracked stereo dis- 
play system of Figure 6. A virtual object 350. a real object 360, and a virtual object 370 are perceived by the viewer 
300. The computer 10 generates the virtual objects 250 and 370 by generating stereo images on the stereo CRT 24, 
to which are reflected by the half silvered mirror 310 as previously discussed. The real object 360 is positioned behind the 
half silvered mirror 310 in relation to the viewer 300. Light reflected from the real object 360 passes through the half 
silvered mirror 31 0, and is perceived by the viewer 300. The computer 1 0 causes the LCD array panel 320 to selectively 
block light reflected from the real object 360. 

[0065] The physical object data base indicates the 3D geometry of the real object 360, and indicates the physical 
75 location in the DPC space of the real object 360. The real object 360 is modeled as a black object in the physical object 
data base. The image rendering system of the computer 10 renders the physical object 360 in the virtual space in order 
to generate z buffer values for the physical object 360. 

[0066] The real object 360 is positioned within the DPC space in front of the virtual object 370 from the point of view 
of the viewer 300. As illustrated, the z buffer of the image rendering system of the computer 1 0 causes no light to be 
20 emitted from the stereo CRT 24 for portions of the virtual object 370 obscured by the real object 360. Thus, the viewer 
300 perceives the virtual object 370 to be positioned behind the real object 360. 

[0067] The real object 360 is positioned within the DPC space behind the virtual object 350 from the point of view 
of the viewer 300. As illustrated, the computer 10 causes the pixels of LCD array panel 320 corresponding to portions 
of the physical object 360 that are behind the virtual object 350 to block light from the physical object 360. Light from 
25 the stereo images on the stereo CRT 24 is reflected by the half silvered mirror 310. As a result, the viewer 300 perceives 
the virtual object 350 to be positioned in front of the real object 360. 

[0068] The real object 360 positioned behind the mirror 310 appears to be in front of the virtual object 370 if a model 
of the real object 360 has been rendered within the DPC space at the proper distance, size, and orientation, with a com- 
pletely black color. Rendering of the real object 360 in the DPC space causes no light to be emitted from the stereo CRT 
so 24 (beyond the minimum black level). As a consequence, the only changing light coming from the portion of the viewers 
gaze corresponding to the real object 360 is the light from the real object 360. 

[0069] The virtual object 350 appears to be in front of the real object 360, but only as a transparent object unable 
to occlude the images of the real object 360. The range of transparency is limited by the configuration of the mirror 310. 
The virtual object 350 exhibits the least amount of transparency when rendered at full intensity. The degree of transpar- 

35 ency depends upon the percentage of reflected versus transmitted light from the mirror 310. The virtual object 350 
exhibits higher degrees of transparency for any particular mirror when rendered at a lower intensity. 
[0070] The LCD array panel 320 enables the computer 10 to control the amounts of transmitted versus reflected 
light perceived by the viewer 300. The LCD array panel 320 is positioned on the far side of the half silvered mirror 310 
from the viewer 300. The LCD array panel 320 controls the amount of directly transmitted light from the real object 360 

ao to the viewer 300. The LCD array panel 320 controls transfer of light on a per pixel basis. 

[0071] The LCD array panel 320 controls light transfer in two modes. In binary mode, the LCD array panel 320 
transfers all or none of the light from real object 360 on a per pixel basis. In "alpha" mode, a percentage of light from 
the real object 360 is transferred. The percentage of light transferred is set to a fractional amount per pixel (within the 
LCD's dynamistic range, update rates, etc.) The binary mode is an extreme version of the alpha mode in which the con- 

45 trol value "alpha" can take on only the values of 0 or 1 . 

[0072] In binary mode, the virtual object 350 can occlude the real object 360. To accomplish this, an alpha frame 
buffer storage bit is maintained by the computer 10 for each pixel of the LCD array panel 320. When rendering the vir- 
tual object 350, the alpha values corresponding to the virtual object 350 are set to 1 (i.e. no physical object light passes 
through the LCD array panel 320). The alpha values corresponding to the "black" model of the real object 360 are set 

so to 0. 

[0073] When z-buffering is performed by the computer 10 for the virtual object 350 and the real object 360, the clos- 
est object type (real or virtual) at each pixel will set the alpha bit to the proper value as outlined above. The alpha image 
is displayed on the LCD array panel 320, causing the proper light vaiving effects. A form of transparency is rendered for 
both real and virtual objects by "screen<loor" transparency methods. 
55 [0074] In alpha mode, transparency is performed by controlling the amount of light passing through each pixel of 
the LCD array pane! 320. The computer 10 implements a rendering method that sets the appropriate fractional alpha 
value in the frame buffer to control the transparency of the LCD array panel 320. The alpha mode enables the viewer 
300 to perceive a transparent virtual object 350 in front of the real object 360 (solid or transparent). The alpha mode 
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also enables the viewer 300 to perceive a transparent real object 360 (like a glass) in front of the virtual object 370 (solid 
or transparent). 

[0075] The value chosen for alpha may be a constant tor a given object, or a complex function computed according 
to the real or simulated optical properties of the real or virtual object, respectively. Complex effects, such as the viewer 
300 perceiving a magnified virtual object through a real magnifying glass can be performed. Alternatively, the real 
objects can be digitized and transferred to the computer 10, such that the computer 10 performs the appropriate optical 
simulations and effects. The computer 10 completely replaces a physical object with a doppenganger virtual object. 
Thus, in the apparatus of Figure 6, some physical objects have pure black virtual counterparts, while some physical 
objects are replaced with non-black virtual copies by setting the corresponding alpha values to 1 . 
[0076] In the foregoing specification, the invention has been described with reference to specific exemplary embod- 
iments thereof. The specification and drawings are accordingly to be regarded as illustrative rather than restrictive. 

Claims 

15 1. An apparatus for generating an accurate stereo three dimensional image, comprising: 

a display device (24) having a display surface (55) for generating stereo images rendered by a computer 

means (10), 

and characterised by: 

20 reflector means (54) coupled to rotate around the display surface (55) of the display device (24) and reflect the 

stereo images, such that the reflector means (54) transmits the stereo images to a viewer, 
means for positioning (100) the reflector means (54) under control of the computer means (10), such that the 
reflector means (54) tracks movement of the viewer, thereby continuously transmitting the stereo images to the 
viewer. 

25 

2. The apparatus of claim 1 , wherein the display device (24) comprises a stereo CRT 

3. The apparatus of claim 1 , wherein the reflector means (54) comprises a mirror, the mirror positioned to form a sub- 
stantially forty-five degree angle in relation to the display surface (55) of the display device (24) as the mirror rotates 

30 around the display surface. 

4. The apparatus of claim 3, wherein the mirror comprises a half silvered mirror. 

5. The apparatus of claim 1, wherein the means for positioning (140) the reflector means (54) under control of the 
35 computer means comprises: 

means for sensing (142) a three dimensional position of the eyes of the viewer; 

motor means (30) coupled to move a rotating platform (56), the reflector means (54) coupled to the rotating 
platform (56) the motor means (30) rotating an angular position of the rotation platform (56) to track the three 
<0 dimensional position of the eyes of the viewer. 

6. The apparatus of claim 1 , further comprising means for moving (30) the display device (24) and the reflector means 
(54) along a two dimensional plane under control of the computer means (10) such that the display device (24) and 
the reflector means (54) 'track the viewer and transmit the stereo images to the viewer. 

45 

7. A method for generating an accurate stereo three dimensional image, comprising the step of: 

generating stereo images on a display surface (55) of a display device (24), the stereo images rendered by a 
computer means (10); 
so and characterised by the steps of : 

reflecting the stereo images with a reflector means (54) coupled to rotate around the display surface (55) ol the 
display device (24) such that the reflected stereo images are transmitted to a viewer; 
positioning the reflector means (54) under control of the computer means (10) such that the reflector means 
(54) tracks movement of the viewer, thereby continuously transmitting the stereo images to the viewer. 

55 

8. The method of claim 7, wherein the display device (24) comprises a stereo CRT. 

9. The method of claim 7, wherein the reflector means (54) comprises a mirror, the mirror positioned to form a sub- 
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stantiafly forty-five degree angle iri relation to the display surface (55) of the display device (24) as the mirror rotates 
around the display surface (55). 

10. The method of claim 7, wherein the step of reflecting the stereo images comprises the step of reflecting the stereo 
5 images in a half -silvered mirror. 

1 1 . The method of daim 7, wherein the step positioning the reflector means (54) under control of the computer means 
(10) comprises the steps of: 

10 sensing a three-dimensional position of the eyes of the viewer (140); 

rotating an angular position of the reflector means to track the three dimensional position of the eyes of the 
viewer (110); 

1 2. The method of claim 7, further comprising the step of moving the display device (24) and the reflector means (54) 
is along a two dimensional plane under control of the computer means (1 10), such that the display device (24) and 

the reflector means (54) track the viewer and transmit the stereo images to the viewer. 

Patentanspruche 

20 1. Eine Einrichtung zum Erzeugen eines genauen dreidimensionalen Stereobildes, aufweisend: 

eine eine Anzeigeoberflache (55) aufweisend e Anzeigeeinrichtung (24) zum Erzeugen von durch eine Compu- 
tereinrichtung (10) hergestellten Stereobildern. 
und gekennzeichnet durch; 

25 Reflektormittel (54). die so angekoppeft sind, daB sie urn die Anzeigeoberflache (55) der Anzeigeeinrichtung 

(24) rotieren und die Stereobilder reflektieren, so daB die Reflektormittel (54) die Stereobilder an einen 
Betrachter Qbermitteln, 

Mittel zum Positionieren (100) der Reflektormittel (54) unter Steuerung der Computereinrichtung (10) derail, 
daB die Reflektormittel (54) der Bewegung des Betrachters nachgefuhrt werden, wodurch die Stereobilder 
30 dem Betrachter kontinuierlich ubermittelt werden. 

2. Die Einrichtung nach Anspruch 1 , wobei die Anzeigeeinrichtung (24) eine Stereo-CRT aufweist. 

3. Die Einrichtung nach Anspruch 1, wobei die Reflektormittel (54) einen Spiegel umfassen, wobei der Spiegel so 
35 positioniert ist. daB er einen Winkel von im wesentlichen 45° in Bezug aul die Anzeigeoberflache (55) der Anzeige- 
einrichtung (24) bildet, wenn der Spiegel urn die Anzeigeoberflache rotiert. 

4. Die Einrichtung nach Anspruch 3, wobei der Spiegel ein halb versilberter Spiegel ist. 

40 5. Die Einrichtung nach Anspruch 1 , wobei die Mittel zum Positionieren (140) der Reflektormittel (54) unter der Steue- 
rung der Computereinrichtung umfassen: 

Mittel zum Erfassen (142) einer raumlichen Position der Augen des Betrachters; 

Motormittel (30), die so -angekoppeft sind, daB sie eine rotierende Plattform (56) bewegen, wobei die Reflek- 
45 tormittel (54) mit der rotierenden Plattform (56) gekoppelt sind, wobei die Motormittel (30) eine Winkel position 

der rotierenden Plattform (56) drehen, um die raumliche Position der Augen des Betrachters zu verfolgen. 

6. Die Einrichtung nach Anspruch 1 , ferner umfassend Mittel zum Bewegen (30) der Anzeigeeinrichtung (24) und der 
Reflektormittel (54) entlang einer zweidimensionalen Ebene unter der Steuerung der Computereinrichtung (10) 

so derart, daB die Anzeigeeinrichtung (24) und die Reflektormittel (54) den Betrachter verfolgen und die Stereobilder 
an den Betrachter Qbermitteln. 

7. Ein Verfahren zum Erzeugen eines genauen dreidimensionalen Stereobildes, umfassend die Schritte: 

55 Erzeugen von Stereobildern auf der Anzeigeoberflache (55) einer Anzeigeeinrichtung (24), wobei die Stereo- 

bilder durch eine Computereinrichtung (10) wiedergegeben werden; 
und gekennzeichnet durch die Schritte: 

Reflektieren der Stereobilder mit einem Reflektormittel (54), das so angekoppeft ist, daB es sich um die Anzei- 
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geoberf lache (55) der Anzeigeeinrichtung (24) derart dreht. da8 die reflektierten Stereobitder an einen 
Betrachter ubermittelt werden; 

Positionieren der Reflektormittel (54) unter der Steuerung der Computereinrichtung (10) derart, daB die 
Reflektormittel (54) die Bewegung des Betrachters verfolgen, wodurch die Stereobilder kontinuierlich dem 
5 Betrachter ubermittelt werden. 

8. Das Verfahren nach Anspruch 7, wobei die Anzeigeeinrichtung (24) eine Stereo-CRT ist 

9. Das Verfahren nach Anspruch 7, wobei die Reflektormittel (54) einen Spiegel aufweisen, wobei der Spiegel so 
10 positioniert wird, daB er zu der Anzeigeoberf lache (55) der Anzeigeeinrichtung (24) einen Winke! von im wesentti- 

chen 45° bildet, wenn der Spiegel sich urn die Anzeigeoberf lache (55) dreht. 

10. Das Verfahren nach Anspruch 7, wobei der Schrrtt des Reflektierens der Stereobilder den Schritt des Reflektierens 
der Stereobilder an einem halb^versilberten Spiegel umfafJt. 

is 

11. Das Verfahren nach Anspruch 7, wobei der Schritt des Positionierens der Reflektormittel (54) unter der Steuerung 
der Computereinrichtung (10) die Schritte umtaBt: 

Erfassen einer raumlichen Position der Augendes Betrachters (140); 
20 Drehen der Reflektormittel urn eine Winkel position, urn die raumliche Position der Augen des Betrachters zu 

verfolgen (110). 

12. Das Verfahren nach Anspruch 7, ferner umfassend den Schrrtt des Bewegens der Anzeigeeinrichtung (24) und der 
Reflektormittel (54) entlang einer zweidimensionalen Ebene unter der Steuerung der Computereinrichtung (1 1 0) 

25 derart, daB die Anzeigeeinrichtung (24) und die Reflektormittel (54) den Betrachter verfolgen und die Stereobilder 
an den Betrachter ubermitteln. 

Revendications 

30 1 . Appareil pour la generation d'une image stereo precise en trois dimensions, comprenant : 

un dispositjf d'atf ichage (24) possedant une surface d'affichage (55) pour la generation d'images stereo visua- 
lisees a I'aide d'un moyen d'ordinateur (10) ; et 
caracterise par ; 

35 

- un moyen de reflecteur (54) couple pour tourner autour de la surface d'affichage (55) du dispositif d'affichage 
(24) et pour reflechir les images stereo de telle facon que le moyen de reflecteur (54) transmette les images 
stereo a un observateur ; et 

40 - un moyen pour positionner (100) le moyen de reflecteur (54) sous la commande du moyen d'ordinateur (10) de 
telle. facon que le moyen de reflecteur (54) suive un deplacement de I'observateur, transmettant ainsi en con- 
tinu les images stereo a I'observateur. 

Appareil selon la revendication 1 , dans fequel le dispositif d'affichage (24) comprend un CRT stereo. 

Appareil seton la revendication 1 , dans lequel le moyen de r6fledeur (54) comprend un miroir, le miroir etant posi- 
tionne pour fatre un angle de sensiblement 45° par rapport a la surface d'affichage (55) du dispositif d'affichage 
(24) tandis que le miroir tourne autour de la surface d'affichage. 

so 4. Appareil selon la revendication 3, dans lequel le miroir comprend un miroir semi-argente. 

5. Appareil selon la revendication 1 . dans lequel le moyen pour positionner (140) le moyen de reflecteur (54) sous la 
commande du moyen d'ordinateur comprend : 

55 - un moyen pour detecter (142) une position en trois dimensions des yeux de I'observateur ; et 

un moyen de moteur (30) couple pour deplacer une plate-forme rotative (56), le moyen de reflecteur (54) etant 
couple a la plate-forme rotative (56), le moyen de moteur (30) faisant tourner la plate-forme rotative (56) sur 
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une position angulaire pour suivre la position en trois dimensions des yeux de I'observateur. 

6. Appareil selon ta revendication 1, comprenant, de plus, tin moyen pour deplacer (30) le dispositif d'affichage (24) 
et le moyen de reflecteur (54) le long d'un plan a deux dimensions sous la command e du moyen d'ordinateur (10) 
de telle facon que le dispositif d'affichage (24) et le moyen de reflecteur (54) suivent I'observateur et lui transmet- 
tent les images stereo. 


7. Procecfe pour la generation d'une image stereo precise en trois dimensions, comprenant I'&ape suivante : 

- la generation d'images stereo sur une surface d'affichage (55) d'un dispositif d'affichage (24), les images ste- 
reo etant visualisees par un moyen d'ordinateur (10) ; et 
caracterise par les etapes suivantes : 

la reflexion des images stereo par un moyen de reflecteur (54) couple pour tourner autour de la surface d'affi- 
chage (55) du dispositif d'affichage (24) de telle facon que les images stereo reflechies soient transmises a un 
observateur ; et 


le positionnement du moyen de reflecteur (54) sous la commande du moyen d'ordinateur (10) de telle facon 
que le moyen de reflecteur (54) suive un deplacement de I'observateur, transmettant ainsi en continu les ima- 
20 ges stereo a I'observateur. 

8. Precede selon la revendication 7, selon lequel le dispositif d'affichage (24) comprend un CRT stereo. 

9. Procede selon la revendication 7. selon lequel le moyen de reflecteur (54) comprend un miroir, le miroir etant posi- 
25 tionne pour faire un angle de sensiblement 45° par rapport a la surface d'affichage (55) du dispositif d'affichage 

(24) tandis que le miroir tourne autour de la surface d'affichage (55). 

10. Proc6de selon la revendication 7, selon lequel l'6tape de reflexion des images stereo comprend une etape de 
reflexion des images stereo sur un miroir semi-argent§. 

30 

11. Proc6d6 selon la revendication 7, selon lequel t'etape pour positionner (140) le moyen de reflecteur (54) sous la 
. commande du moyen d'ordinateur comprend les etapes suivantes : 

la detection d'une position en trois dimensions des yeux de I'observateur (1 40) ; et 

35 

la rotation sur une position angulaire du moyen de reflecteur pour suivre la position en trois dimensions des 
yeux de I'observateur (110). 

12. Procede selon la revendication 7, comprenant de plus une etape de deplacement du dispositif d'affichage (24) et 
40 du moyen de reflecteur (54) le long d'un plan a deux dimensions sous la commande du moyen d'ordinateur (110) 

de telle facon que le dispositif d'affichage (24) et le moyen de reflecteur (54) suivent I'observateur et lui transmet- 
tent les images stereo. 


45 
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